A Model for Learning Words in a Language by Crawling the Web

A model for an Internet web crawler with a very limited vocabulary can be devised to learn most words in the English language. The system will have the ability to read a sentence where only constituents of the sentence are known. In order to achieve this, the system will provide a methodology to resolve ambiguities within the unknown constituent words and parts of speech. The system will include a lexicon of word types, nouns, verbs, adjectives, and adverbs to be learned, seeded with 100 random words. The source material being read need not be domain specific. As the overall lexicon of the system improves and grows, domain-specific and technically advanced jargon can be more readily handled. Some categories of word types such as determiners (the, a, each, all, etc.), conjunctions (and, or, but) and prepositions (by, along, etc.) can easily be exhaustively enumerated. An algorithm will be used in the process of determining the nature of unknown words where a lexicon of English, nouns, verbs and adjectives is large.